-
Notifications
You must be signed in to change notification settings - Fork 208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: multichain-testing #9462
feat: multichain-testing #9462
Conversation
Deploying agoric-sdk with Cloudflare Pages
|
0dd975c
to
ad76fce
Compare
baa17f1
to
4f482a6
Compare
.github/workflows/multichain-e2e.yml
Outdated
on: | ||
# for now, only run on-demand | ||
workflow_dispatch: | ||
pull_request: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO remove this line before merging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you want it to run on every PR? consider running it the same times as the docker build tests (merges to master or with force:integration
enabled)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've updated to the conditions for docker.yml
.. unclear to me if force:integration
will pick this up but agree with the suggested strategy
75928b7
to
1dcc9d4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lots of progress!
Since this is a testing framework, the primary thing is that it works and covers what we expect it to. Secondary is DX (e.g. minimizing surprises) which I think this PR should try to minimize. Third is maintainability, which I think we can punt to a follow-up after we have more experience with the tools
.github/workflows/multichain-e2e.yml
Outdated
on: | ||
# for now, only run on-demand | ||
workflow_dispatch: | ||
pull_request: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you want it to run on every PR? consider running it the same times as the docker build tests (merges to master or with force:integration
enabled)
.github/workflows/multichain-e2e.yml
Outdated
path: ./agoric-sdk | ||
|
||
- name: Enable Corepack | ||
run: corepack enable || sudo corepack enable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pretty sure we should never need sudo
in GH actions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had source from here but since removed
.github/workflows/multichain-e2e.yml
Outdated
runs-on: ubuntu-latest-16core | ||
strategy: | ||
matrix: | ||
node-version: ['18.x'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why mess with a matrix?
const urls = []; | ||
config.chains?.forEach((chain) => { | ||
- urls.push(`${registryUrl}/chains/${chain.name}`, `${registryUrl}/chains/${chain.name}/assets`); | ||
+ urls.push(`${registryUrl}/chains/${chain.id}`, `${registryUrl}/chains/${chain.id}/assets`); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this a bug in starship? if so, can we get a fix upstream?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, there was a breaking change in the 0.2.x helm chart release. I opened an issue: cosmology-tech/StarshipJS#3
const deleteKeys = async (deleteKey: (name: string) => Promise<string>) => { | ||
for (const key of KEYS) { | ||
try { | ||
await deleteKey(key); | ||
} catch (_e) { | ||
// ignore | ||
} | ||
} | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You really want delete to fail silently?
const deleteKeys = async (deleteKey: (name: string) => Promise<string>) => { | |
for (const key of KEYS) { | |
try { | |
await deleteKey(key); | |
} catch (_e) { | |
// ignore | |
} | |
} | |
}; | |
const deleteKeys = (deleteKey: (name: string) => Promise<string>) => | |
Promise.all(KEYS.map(key => deleteKey(key).catch())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was helpful when I was working out the correct pattern here. I've since refactored in support.ts
to:
const makeKeyring: (e2eTools: Pick<E2ETools, "addKey" | "deleteKey">) => Promise<{
setupTestKeys: (keys?: string[]) => Promise<Record<string, string>>;
deleteTestKeys: () => Promise<string[]>;
}>
If a test needs keys it can call setupTestKeys
in test.before()
and .deleteTestKeys()
in test.after()
|
||
const bondDenom = | ||
useChain('osmosis').chain.staking?.staking_tokens?.[0].denom; | ||
if (!bondDenom) throw new Error('Bond denom not found.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (!bondDenom) throw new Error('Bond denom not found.'); | |
t.truthy(bondDenom, 'Bond demon not found'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated to this to appease to the TS compiler:
if (!bondDenom) {
t.fail('Bond denom not found.');
} else {
const { balance } = await queryClient.queryBalance(addr, bondDenom);
t.deepEqual(balance, { denom: bondDenom, amount: '10000000000' });
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here's hoping we get to Ava 6 soon which terminates on failed assertions, thus narrowing the code following
multichain-testing/tools/agd-lib.js
Outdated
|
||
const { freeze } = Object; | ||
|
||
const agdBinary = 'kubectl'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not really :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please rename to be accurate. e.g. kubectlBinary
/** | ||
* @param {{ execFileSync: ExecSync }} io | ||
*/ | ||
export const makeAgd = ({ execFileSync }) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we'll want to DRY this with other helpers and I'd also like to simplify it. but this works for now
export const pathToKey = (path) => path.join('.'); | ||
|
||
/** @param {string} key */ | ||
export const keyToPath = (key) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a little surprised we don't have exports that do this already
09ea9b9
to
8df0cb5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I followed the docs and got it working with a bit of debugging with @0xpatrickdev. Some minor suggestions for the missing bits.
```bash | ||
make fund-provision-poool | ||
``` | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Following this guide everything works except the two steps below. Would be helpful to provide an example:
For the steps below, you must import a key to `agd` or create a new one. For example, we can generate one to use as `ADDR` in the next steps: | |
`kubectl exec -i agoriclocal-genesis-0 -c validator -- agd keys add user1` | |
I considered adding a make target for adding a new key instead of the kubectl command, but not sure if it's useful beyond one-offs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, added this note as a note
multichain-testing/README.md
Outdated
ADDR=agoric123 COIN=100000ubld make fund-wallet | ||
ADDR=agoric123 make provision-smart-wallet |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think because these are make vars, not envvars, they did not work as is in my shell. I think the correct COIN
amount is 10000000ubld
. I needed to instead do:
ADDR=agoric123 COIN=100000ubld make fund-wallet | |
ADDR=agoric123 make provision-smart-wallet | |
make fund-wallet ADDR=agoric123 COIN=10000000ubld | |
make provision-smart-wallet ADDR=agoric123 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch, thank you. I've applied these changes. Also upped to 20000000ubld
so there's 10 BLD left over for gas.
multichain-testing/Makefile
Outdated
|
||
provision-smart-wallet: | ||
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from faucet -y -b block |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This did not work for me because --from
key needs to match ADDR
. We need to pass whatever key name matches ADDR
:
provision-smart-wallet: | |
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from faucet -y -b block | |
KEY_NAME=user1 | |
provision-smart-wallet: | |
kubectl exec -i agoriclocal-genesis-0 -c validator -- agd tx swingset provision-one wallet $(ADDR) SMART_WALLET --from $(KEY_NAME) -y -b block |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks - fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking good. I don't love the copy-pasta but since it does have an issue I agree it's appropriate to punt.
I'm requesting the package.json changes because that's a risk to merge.
Other than that, I judge this mostly by whether it improves the testing situation. It clearly does and I expect we'll continue to iterate.
multichain-testing/package.json
Outdated
"name": "@agoric/multichain-testing", | ||
"version": "0.1.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We shouldn't be publishing this, so it doesn't need a version. We shouldn't be importing from it so it doesn't need a name.
It does need this to prevent publishing:
"name": "@agoric/multichain-testing", | |
"version": "0.1.0", | |
"private": true, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was thinking external developers might want to depend on this at some point, but suppose it could be viewed as boilerplate instead. Can cross that bridge when we come to it.
"prettier": "^3.2.4", | ||
"starshipjs": "2.0.0", | ||
"tsimp": "^2.0.10", | ||
"typescript": "^5.3.3" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please match the rest of the repo
"typescript": "^5.3.3" | |
"typescript": "^5.5.0-beta" |
Not a big deal for the other deps but it'll be surprising to a developer if some code works in one place and not the other
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree it would be nice. 5.5.0-beta
doesn't seem to play nicely here for whatever reason
multichain-testing/package.json
Outdated
"@types/node": "^20.11.13", | ||
"@typescript-eslint/eslint-plugin": "^6.20.0", | ||
"@typescript-eslint/parser": "^6.20.0", | ||
"ava": "^5.3.0", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
consider using Ava 6 at this outset, so there's not more work to migrate later
const osmosisFaucet = useChain('osmosis').creditFromFaucet; | ||
t.log('Requeting faucet funds'); | ||
await osmosisFaucet(addr); | ||
await sleep(1000, t.log); // needed to avoid race condition |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please document with an XXX
to help someone debugging the race or who wants to clean it up
if (!bondDenom) { | ||
t.fail('Bond denom not found.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (!bondDenom) { | |
t.fail('Bond denom not found.'); | |
t.truthy(bondDenom, 'Bond denom not found.'); |
would be more clear. I imagine you used the if
to get type narrowing for the queryBalance
call. This is a reason to adopt Ava 6. IIUC its assertions also do type assertions so lines after t.truthy
will know bondDenom
is not nullish.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ava 6. IIUC its assertions also do type assertions so lines after t.truthywill know bondDenom is not nullish.
No luck here, seemingly. I used the non-null-assertion-operator to make this more readable and drop the if/else
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@@ -0,0 +1,21 @@ | |||
MIT License | |||
|
|||
Copyright (c) 2024 Interweb, Inc. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well done to include
multichain-testing/tools/agd-lib.js
Outdated
|
||
const { freeze } = Object; | ||
|
||
const agdBinary = 'kubectl'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please rename to be accurate. e.g. kubectlBinary
80e010b
to
3b72fef
Compare
multichain-testing/package.json
Outdated
"serial": true, | ||
"environmentVariables": { | ||
"LOCKDOWN_STACK_FILTERING": "verbose" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wondered if we should do this throughout the repo, but I think this accomplishes the same and more:
"serial": true, | |
"environmentVariables": { | |
"LOCKDOWN_STACK_FILTERING": "verbose" | |
} | |
"require": [ | |
"@endo/init/debug.js" | |
], | |
"serial": true |
1efac9c
to
3c959c5
Compare
3c959c5
to
c74d0c5
Compare
- request to http://localhost:8081/chains/agoriclocal failed, reason: connect ECONNREFUSED ::1:8081
2024-04-11 17:25 3c29089
- combine starshipjs utils with e2e smart wallet utils - create patch for js-yaml - apply existing patch for node-fetch and force resolution
c74d0c5
to
54f453f
Compare
closes: #XXXX
refs: #8896
Description
@agoric/multichain-testing
package outside of the yarn workspaceconfig.yaml
includes agoric, osmosis, cosmos, and hermes relayers between each. A chain registry (served over http), faucet, and block explorer are also provided).@agoric/synthetic-chain
anddapp-agoric-basics
that help towards a smart wallet client that can execute offers.Security Considerations
Scaling Considerations
Taking on some tech debt here wrt smart wallet utilities and being DRY, but we plan to address this in future. See #8963
Documentation Considerations
README.md documentation for running the service is provided.
Testing Considerations
The goal of this PR is to build greater confidence in our software via automated testing with fully-simulated chains.
Upgrade Considerations